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Abstract 

With the increasing number of components and fnrther miniaturiza¬ 
tion the mean time between faults in supercomputers will decrease. Sys¬ 
tem level fault tolerance techniques are expensive and cost energy, since 
they are often based on redundancy. Also classical check-point-restart 
techniques reach their limits when the time for storing the system state 
to backup memory becomes excessive. Therefore, algorithm-based fault 
tolerance mechanisms can become an attractive alternative. This article 
investigates the solution process for elliptic partial differential equations 
that are discretized by finite elements. Faults that occur in the parallel 
geometric multigrid solver are studied in various model scenarios. In a 
standard domain partitioning approach, the impact of a failure of a core 
or a node will affect one or several subdomains. Different strategies are 
developed to compensate the effect of such a failure algorithmically. The 
recovery is achieved by solving a local subproblem with Dirichlet bound¬ 
ary conditions using local multigrid cycling algorithms. Additionally, we 
propose a superman strategy where extra compute power is employed to 
minimize the time of the recovery process. 
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1 Introduction 

Future high performance systems will be characterized by millions of compute 
nodes that are executing up to a billion of parallel threads. This compute 
power will be expensive due to the associated investment and operational cost 
which will include energy consumption as a rapidly increasing factor. Hardware, 
software and algorithmic components of such a large scale computing are inter¬ 
dependent, and thus reliability of each single component is necessary. Since this 
increasing complexity results in a higher probability of any kind of failure in the 
HPC-system [8], strategies which circumvent and/or accomplish such a behavior 
are inevitable. A commonly used approach is copying the data to backup disks 
on the I/O-system and restart the run with the stored data in case of a failure. 
These checkpoint/restart strategies need to collect and transfer data from and 
to all processors and are, in general, too costly and not attractive. 

Geometric multigrid methods can deliver an asymptotic optimal complexity 
and can be implemented with excellent efhciency on large scale parallel machines 
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[2, 19, 18, 30]. Typical future runs involving multigrid computations will last 
from a few hours to weeks and use up to a billion threads. Therefore, an error- 
resilient methodology for any failure will be required and necessary to obtain 
fault-free and efficiently computed solutions. 

The type of failures and their treatment can be categorized in hardware- 
based fault tolerance (HBFT) [25, 17, 26, 24], software-base fault tolerance 
(SBFT) [5, 31, 16, 3] and algorithm-based fault tolerance (ABFT) [22, 7, 11], 
for a general overview of fault tolerance techniques we refer to [8, 9, 10]. 

In this paper, we focus on the algorithmic approaches for handling failures. 
ABFT improves the reliability of the HPC-system in detecting the failure and 
correcting the results by implementing the resilience in the algoithm itself. Orig¬ 
inally, ABFT was proposed by Huang and Abraham [22] for systolic arrays where 
due to checksums the persistency of the data involved in the algorithm is moni¬ 
tored and reconstructed. Later, it was extended to applications in linear algebra 
such as addition, matrix operations, scalar product, LU-decomposition, trans¬ 
position and in fast Fourier transformation [1, 4, 13, 23]. Currently, the work by 
Davies and Chen [15] efficiently deals with fault detection and correction during 
the calculation for dense matrix operations. For iterative - linear and Krylov 
space - solvers for sparse matrix such as SOR, GMRES, CG-iterations the previ¬ 
ous mentioned approaches are not suitable, since this can result in high overhead 
for sparse linear algebra [28] and were consequently adapted by [7, 12, 27, 29]. 
Cui et al. propose in [14] a technique to use the structure of a parallel subspace 
correction method such that the subspaces are redundant on different processors 
and the workload is efficiently balanced. Further, an algebraic multigrid solver 
was analyzed in [11] where the most vulnerable components are identified and 
resilience techniques are explored. 

Here, we investigate in a fault tolerant parallel geometric multigrid method. 
Similar to [14] we pursue fault tolerant strategies which 

• converge when a fault occurs assuming it is detectable, 

• minimize the delay in the solution process, 

• minimize computational and communication overhead. 

In order to achieve these goals, we study the consequences of failures for a 
geometric multigrid algorithm from bottom up. By applying local correction 
methods, we recover a partial solution and use it such that the effect of the 
fault on the global solution procedure is minimized. The major difference to 
other approaches in ABFT [11, 14] is that we proceed without check-pointing 
data of the solution but rather recalculate the faulty part. 

Our paper is organized as follows: In Sec. 2, we describe the model equation 
and briefly discuss the parallel hierarchical hybrid multigrid (HHG) framework 
that will serve as the basis of the study in this paper. Next, we introduce the 
failure scenario that is used to study the influence of a fault within a parallel 
geometric multigrid method and its effect on the convergence. In Sec. 3, we then 
develop local recovery strategies and demonstrate by numerical experiments how 
these improve the recovery behavior after the fault. 
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2 Faulty Solution Process 

2.1 Model problem and geometric multigrid 

This paper considers, for simiplicity of notation, the Laplace equation with 
Dirichlet boundary conditions 

—Aw = 0 in n, u = g on dft (1) 

as model problem for the design and analysis of a fault recovery algorithm. 
Here, C is a bounded polyhedral domain. is triangulated with an 



Fig. 1: Structured tetrahedral refinement. 

unstructured tetrahedral mesh that we denote by 1^-2- From this initial coarse 
mesh, a hierarchy of meshes T := {71, Z = 0,..., L} is constructed by successive 
uniform refinement as illustrated in Fig. 1. 

The discretization of (1) uses conforming linear finite elements (FE) on Ti 
that leads canonically to a nested sequence of finite element spaces Vb C Vi C 
... C Vz, C H^{n) and a corresponding family of linear systems 

Aiui=l- l = 0,...,L. (2) 

The Dirichlet boundary conditions are included in the linear systems (2). 

This hierarchy will be used to set up an iterative multigrid solver and to 
define the error recovery strategy. Multigrid methods can achieve level-inde¬ 
pendent convergence rates with optimal complexity 0{N), where N is the num¬ 
ber of unknowns, cf. [6, 20]. We apply multigrid correction schemes in V-, W-, 
or F-cycles with standard components to (2). Explicitly, we use linear transfer 
operators and a hybrid variant of a Gauss-Seidel updating scheme as smoother. 

2.2 Hierarchical Hybrid Grids 

The Hierarchical Hybrid Grids (HHG) framework [2, 19, 18] is designed to com¬ 
bine the flexibility of unstructured FE meshes with the performance advantage 
of structured grids in a block-structured approach. 

The implementation is based on domain partitioning that splits the mesh 
into primitives: vertices, edges, faces, and volumes. In the HHG data structure 
each primitive is then refined regularly resulting in a global block-structured 
grid. For our later error recovery strategies, the domain partitioning approach 
is crucial, but the block-structured mesh structure could be generalized to fully 
unstructured meshes. The multigrid operations such as smoothing, prolonga¬ 
tion, restriction, and residual calculation, are exploited such that they typically 
operate on the primitive itself and its neighboring primitives via ghost layers. 


3 








These operations are inherently local and suited for parallel computations on a 
distributed memory system using message passing with MPI. Here, the prim¬ 
itives are mapped to processors that execute the local operations. The data 
dependencies require a systematic exchange of the ghost layers. This func¬ 
tionality is provided in a transparent and highly optimized form in the HHG 
framework. 

2.3 Fault Model 

We assume that a failure in the solution process for (2) can occur at any time. 
For our study, we concentrate on a specific fault model under assumptions sim¬ 
ilar to [14, 21]. We restrict the analysis, for simplicity, to the case that only 
one process crashes. All strategies can be extended easily to a defect of more 
processors, since they only rely on the locality of the fault. 

Furthermore, we concentrate on the case of using V-cycles for the solution 
of (2). The input tetrahedral mesh T -2 defines the partitioning used for paral¬ 
lelization in HHG. Each tetrahedron in T -2 is mapped to a processor, including 
all the refined subdomain meshes contained in the coarsest level element. Gon- 
sequently, the number of subdomains and the number of processes is equal to 
the number of tetrahedra in T- 2 - 

If a process experiences a fault, the information of the subdomain is lost. In 
the context of this article, the faulty subdomain Up G Q is just a single tetra¬ 
hedron in T- 2 - The other tetrahedra constitute the healthy subdomain VLh, i.e. 
those tetrahedra that are not affected by the fault. Healthy and faulty regions 
are separated by an interface F/ := dVln H dVlp. In the finite element mesh, as 
implemented in HHG, the interface region F/ contains the nodes living on faces, 
edges, and vertices of the input mesh. These data structures are responsible for 
handling communication in HHG and are thus stored redundantly in the form 
of ghost instances on several processors. Thus, even if one of the instances is 
lost due to the fault, a complete recovery is always possible for them, and thus 
we assume implicitly that the data associated with them are unaffected by the 
fault. 

In Fig. 2, the setup is illustrated for a computational domain with 16 mil¬ 
lion unknowns. The domain consists of 48 tetrahedral subdomains that are 
distributed to 48 processors. Then each subdomain includes 300 000 unknowns 
and, thus, the failure of a process causes the loss of information for 300 000 
unknowns. 




Fig. 2: Fault of one process. Left: Gomputational domain (here: 16 mil. un¬ 
knowns) with a faulty (red) subdomain, one input grid tetrahedron (here: 300 
000 unknowns). Right: Faulty subdomain. 
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For our strategy, it is necessary that we can detect erroneous processes 
quickly and then adapt the solution procedure dynamically. Unfortunately, the 
current supercomputer systems and the fault tolerant MPI-extensions such as 
Harness FT-MPI ^ or ULFM ^ do not yet support this functionality as ideally 
needed. For the purposes of this study, we suppose a failure is reported as 
soon as it occurs during a multigrid cycle. When a process crashes, we assign 
a new - until then not used - substitute process to take over its job. This as¬ 
sumes that a large scale parallel computation is started with a certain number 
of substitute processors initially being idle - very much like substitute players 
in a team sport match. The solution values in the faulty subdomain are set to 
zero as initial guess. Other initial guesses could also be used, such as data from 
previous check-pointing or values obtained by interpolation, but this will not 
be considered here in detail. After the local re-initialization of the problem, we 
continue with multigrid cycles in the solution process. 



Fig. 3: Cross section through the domain fl and the surface of the faulty tetrahe¬ 
dron. Left figure: residual error directly after the failure. Right figure: residual 
error after one additional global V-cycle. a := \ogiQ{\Residual\). 

In a first experiment, we consider the performance of our multigrid iteration 
when it is continued after a fault has occurred. All numerical experiments are 
performed within the HHG framework introduced in Subsec. 2.2. We choose the 
computational domain U = (0,1)^, and g = sin(7r(a; -|- V^y)) sinh(-\/37rz) in (1) 
with the described setup in Fig. 2. In the solution process, we apply V-cycles 
with three pre- and post-smoothing steps of the Gauss-Seidel smoother of Sec. 
2 . 1 . 

In Fig. 3, the residual is visualized on a cross section through the domain 
together with the surface of the tetrahedron where the fault had occurred af¬ 
ter 5 global iterations. Right after the failure and after re-initialization, the 
largest residual error is clearly located in this tetrahedron. These large local 
error components are transported over the whole domain in the course of the 
following multigrid iterations. Though each application of the smoother trans¬ 
ports information only across a few neighboring mesh cells, multigrid employs 
coarser grids recursively. The smoothing on these grids leads to the global data 
exchange that is essential for the level-independent convergence of multigrid it¬ 
erations. Therefore, though the residual is reduced efficiently (in the L 2 -norm) 
by such iterations, we observe a pollution of the residual error across the whole 
domain. This can be seen in Fig. 3 at the right, where the overall residual error 
has been reduced after the additional V-cycle (note the scaling), but the error 
now pollutes the whole domain. 

^http://id.cs.utk.edu/ftmpi/ 

^http: //fault-tolerance.org/ 


5 





The numerical behavior is analysed quantitatively in Fig. 4. After the 
fault the residual norm jumps from 6.24 • 10“^ up to 1.55 • 10“^. If a complete 
checkpointing-recovery (CCR) of the lost values could be performed, it would 
fully restore the residual from before the fault. Note, that this recovery, as 
marked in the diagram with no fault, introduces no additional computational 
effort in comparison to the situation without failure, but writing and reading 
checkpoint data would be too expensive for large scale computations. However, 
the failure introduces error components that can be reduced efficiently by the 
multigrid method as can be seen in the residuals marked with fault. In the 
first cycles after the fault, we observe a pre-asymptotic convergence rate that is 
better than the asymptotic rate for roughly three cycles. This helps significantly 
to compensate for the fault. The roundoff error limit of approximately 10“^® 
is reached after a total of 20 V-cycles, as compared to 16 V-cycles that were 
necessary in the unperturbed computation. As expected these effects can be 



Iterations 


Fig. 4: Convergence of the residual error scaled by the initial residual with fault 
after 5 iterations. 

seen more drastically, when the fault occurs at a later step during the iteration 
process. The situation of a fault after 7 iterations is displayed in Fig. 5 (left) 
and after 11 iterations in Fig. 5 (right). In those cases, the global residual is 
already quite small when the fault occurs, and we need 7 and 10 more iterations, 
respectively, to obtain the rounding error limit of 10“^®. 




Iterations Iterations 


Fig. 5: Convergence of the residual error scaled by the initial residual with fault 
after 7 iterations (left) and 11 iterations (right). 
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3 Local Recovery Strategy 

Avoiding the global pollution observed in Subsec. 2.3 motivates a recovery strat¬ 
egy. For the recovery step, we here require that it is local, i.e., can be performed 
without communication. For the overall efficiency of the recovery strategy, it is 
essential that it can be performed as quickly as possible. We therefore propose 
a superman strategy, which simply means that more resources are devoted to 
perform the recovery in an attempt to reduce the recovery time. This becomes 
especially attractive, since the recovery procedure is local, i.e. the superman 
substitute processor can focus its attention on a single subdomain. Note, that 
during the local recovery, the global solution process cannot continue unaffected, 
since this would require to use values from the faulty subdomain. This access to 
the faulty subdomain must be prevented, since otherwise a global error pollu¬ 
tion would occur. The values in the faulty subdomain will only become available 
again, when the local recovery has been completed. 

Technically the speedup of the superman strategy can be accomplished by 
additional parallelization. We propose here, that e.g. a full (shared memory) 
compute node is assigned to perform the local recovery for a domain that was 
previously handled by a single core. This can be accomplished by a dedicated 
OpenMP parallelization for the recovery process. Otherwise, of course, a fur¬ 
ther domain partitioning of the faulty region might be used together with an 
additional local distributed memory parallelization. 

We denote the speedup factor that is achieved with a superman strategy 
for the local recovery by ?7speedup ^ [IjCxd); if ’Ispeedup “ f’ there is no 
speedup. For the case ??speedup recovery would cost no time. For the 

moment, let us assume such a perfect superman and that the local recovery step 
is in this sense free. Let us define the following local subproblem 

—Au = 0 in Qp, (3) 

with Dirichlet boundary conditions on F/. 

Under the assumptions of Subsec. 2.3, we set the lost values in the faulty 
region flp d to zero and before continuing with global problem (1) we solve 

(3). 

The subproblem (3) can in principle be solved by any method, e.g., the 
relaxation that is used as multigrid smoothing, a direct solver, Krylov space 
iterations, or multigrid cycles. We denote by kp the number of local solver iter¬ 
ations. After solving the subproblem (3) with sufficient accuracy, the solution 
process for the global problem (1) can be resumed. The procedure is presented 
in Alg. 1. 

To quantify the number of multigrid cycles that are saved by using the local 
recovery strategy, we introduce a reference parameter Cycle Advantage which 
we denote by n. We assume a process experiences a fault after k cycles. Then, 
we evaluate the residual after K cycles with k < K. We choose K such that 
the stopping criteria is fulfilled in the no-fault case. Here, and denote 
the residual after a local recovery with kp iterations and when no-local-recovery 
{kp = 0) has been performed, respectively. 

Then, it holds 

I|r^""^ll<ll4°^ll, (4) 
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Algorithm 1 Local recovery algorithm 
1; Solve (1) by multigrid cycles. 

2: if Fault has occurred then 
3: STOP solving (1). 

4: Recover boundary values F/ by neighboring subdomains. 

5: Set values of the defect subdomain to zero. 

6: Solve (3) in Oi? subject to Dirichlet boundary conditions on F/ using 

7; kp local solver iterations with speedup factor ?7speedup- 
8; RETURN to step 1 with new computed values in Vlp. 

9; end if 


since the local strategy improves the residual error. In the case of no-local- 
recovery, we need to apply additionally k cycles such that 




4Lii = 


4>i 


(5) 


where /x is the convergence rate of one multigrid cycle. Thus, we solve for k in 

(5) 

K = log(||r^^^||/||r^^||)/log(Ai). (6) 

Note, that for the calculation of the reference parameter k we need two versions 
of computation runs, one with a fault and a local recovery strategy and one with 
a fault and without a local recovery strategy. A local recovery strategy with 
K = 0 does not improve the residual error in comparison to no recovery strategy, 
whereas a higher k implies an improvement of the residual error of magnitude 
K in multigrid cycles. For example, if k = 5, then, the residual error without 
a local recovery needs 5 additional cycles to achieve the same residual error as 
the local recovery strategy. 

Let us consider again the example of Subsec. 2.3. We study five different 
local recovery strategies: Gauss-Seidel smoothing (Smth), Jacobi preconditioned 
CG (PGG) iterations, local V-cycles, local W-cycles and local F-cycles. 

In Fig. 6, we present three cross sections through the computational domain 
together with the surface of the faulty tetrahedron. In the left plot, the residual 
error directly after the failure is shown, in the middle plot the residual error 
after the local recovery with one F-cycle and in the right plot after an additional 
global V-cycle after the local recovery. We observe two major advantages of local 
recovery strategies compared to no-local-recovery: the local recovery reduces the 
residual error in the defective tetrahedron (middle plot of Fig. 6) and the error 
pollution is much smaller over the computational domain (right plot of Fig. 6). 

In Fig. 7, we consider different local recovery strategies and their impact on 
the solution process. We observe that all local strategies improve the residual 
error directly after the fault in comparison to using no-local-recovery. The 
different cycles, local V- (green), local W- (purple), and local F-cycle (orange) 
strategies, produce almost the same effect on the residual errors. However, using 
10 PGG-iterations (pink) or 10 Gauss-Seidel smoothing steps (light blue) result 
only in an almost negligible improvement compared to using no recovery at 
all. After the fault and recovery, all recovery strategies using a local multigrid 
solver exhibit a favorable pre-asymptotic convergence in the first three global 
iterations after recovery and, then, align with the convergence of GGR. The 
residual error of the smoothing or PGG strategy after three further iterations is 









Fig. 6: Cross section through the domain and the surface of the erroneous 
tetrahedron. Left plot: residual error directly after the failure. Middle plot: 
residual error after applying a local recovery strategy (one local F-cycle). Right 
plot: residual error after an additional V-cycle. a \ogiQ{\Residual\). 
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Fig. 7: Convergence of the residual error scaled by the initial residual for differ¬ 
ent local recovery strategies. 


similar to the approach without a local strategy and, therefore, needs as many 
iterations as the no-local-recovery strategy to reach the prescribed accuracy of 
10“^^, i.e., three additional iterations in comparison to the CCR strategy. Two 
local V-cycles improve the situation such that the delay in finding the solution is 
reduced to one iteration. A local correction by two W-cycles or F-cycles deliver 
almost the same residual error and reduces the delay in comparison to the two 
local V-cycles strategy. A F-cycle is preferred, since its computational cost is 
by a factor of 8/7 lower than for a W-cycle. Further in Fig. 8, we vary the 
number of V-cycles kp for solving the subproblem on Vlp. The delay in finding 
the solution significantly depends on how accurate the subproblem is solved. 
One local V-cycle reduces the solution process by one iteration in comparison 
to no-local-recovery strategy, two V-cycles by three iterations and three V-cycles 
completely compensate the fault. Additionally, we compare the size of k for 
different local recovery strategies and for different iterations after which a failure 
occurs. We study faults after 5, 7, 11 iterations, respectively, and smoothing, 
PCG-iterations, and multigrid V-, W-, and F-cycle as local recovery strategies. 
The V-cycle convergence rate has been numerically evaluated as n — 0.194. The 
cycle advantage n is measured after 15 iterations (K = 15) and presented in 
Tab. 1. 

The residual error of the CCR strategy is of order 10“^"* after 15 iterations 
such that the convergence does not saturate due to round off errors. We need 
4.164, 6.445 or 10.999 additional V-cycles for the case of no-local-recovery and 
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Fig. 8: Convergence of the residual error scaled by the initial residual for differ¬ 
ent number kp of local V-cycles. 


when the failure occurs after 5, 7 or 11 iterations, respectively, in order to 
achieve the same accuracy as in the CCR strategy. This is due to the error 
difference, introduced by the fault, before and after the fault which is larger the 
later it occurs. These are also the maximal n values which can be achieved by 
a local recovery strategy. For the fault after 5 iterations in Tab. 1, we obtain 
similar k for a local recovery with 4 V-cycles, 3 W-cycles, or 3 F-cycle which 
is almost as good as in the CCR strategy. Further, for smoothing and PCG 
iterations, only marginally small improvement can be observed in comparison 
to using multigrid cycles or the CCR strategy. As expected for the fault after 7 
iterations, 5 W- or F-cycles and for the fault after 11 iterations, more than 5 W- 
or F-cycles are necessary to achieve a k similar to the CCR strategy. For the 
other strategies (smoothing and PCG iterations) no significant improvement is 
obtained. Again, we observe that W-cycles and F-cycles yield similar results, 
thus, F-cycles are preferred to W-cycles due to computational cost arguments. 

4 Conclusion and Outlook 

This paper gives a first insight in constructing a fault tolerant multigrid solver. 
It is shown that geometric multigrid solvers are inherently suitable to deal with 
failures of processes. The failure results in a loss of the values of a subdomain. 
To recover these lost values local subproblems with Dirichlet boundary condi¬ 
tions are solved by various strategies ranging from relaxation scheme, Krylov 
space methods to multigrid cycles. Further, the local problems are accelerated 
by a superman strategy through additional parallelization of the recovery. We 
introduce the reference parameter Cycle Advantage which gives the possibility to 
demonstrate to which extent a recovery strategy improves the time-to-solution 
in terms of cycle applications in comparison to use no-recovery. Only multigrid 
cycles can efficiently treat the local subproblem and even reach the same accu¬ 
racy as the complete check-pointing recovery but with a minimized access to 
backup memory. 

In future work, we will extend this basic idea for problems with more un¬ 
knowns, incorporate the local strategy in a global recovery strategy to balance 
the waiting time for all processors. We will enhance our considerations to prob¬ 
lems with non-vanishing right-hand side and analyze its recovery. We will fur¬ 
ther develop acceleration strategies for the local recovery and include them in 
the HHG framework with a fault tolerant MPI-variants. 
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Tab. 1: Cycle Advantage for an early (after 5 iterations) , middle (after 7 itera¬ 
tions) , and late (after 11 iterations) fault for different local recovery strategies. 


Fault After 5 Iter. 

Fault After 7 Iter. 

Fault After 11 Iter. 

Strategies 

K. 

Strategies 

K 

Strategies 

K 

CCR 

4.164 

CCR 

6.445 

CCR 

10.999 

1 X Vcycle 

1.637 

1 X Vcycle 

1.646 

1 X Vcycle 

1.678 

2 X Vcycle 

3.075 

2 X Vcycle 

3.111 

2 X Vcycle 

3.225 

3 X Vcycle 

4.069 

3 X Vcycle 

4.445 

3 X Vcycle 

4.643 

4 X Vcycle 

4.165 

4 X Vcycle 

5.693 

4 X Vcycle 

5.970 

5 X Vcycle 

4.165 

5 X Vcycle 

6.397 

5 X Vcycle 

7.242 

1 X Wcycle 

1.825 

1 X Wcycle 

1.780 

1 X Wcycle 

1.747 

2 X Wcycle 

3.425 

2 X Wcycle 

3.404 

2 X Wcycle 

3.380 

3 X Wcycle 

4.147 

3 X Wcycle 

4.967 

3 X Wcycle 

4.951 

4 X Wcycle 

4.164 

4 X Wcycle 

6.249 

4 X Wcycle 

6.458 

5 X Wcycle 

4.165 

5 X Wcycle 

6.444 

5 X Wcycle 

7.911 

1 X Fcycle 

1.828 

1 X Fcycle 

1.781 

1 X Fcycle 

1.748 

2 X Fcycle 

3.426 

2 X Fcycle 

3.405 

2 X Fcycle 

3.381 

3 X Fcycle 

4.147 

3 X Fcycle 

4.967 

3 X Fcycle 

4.951 

4 X Fcycle 

4.164 

4 X Fcycle 

6.249 

4 X Fcycle 

6.458 

5 X Fcycle 

4.165 

5 X Fcycle 

6.444 

5 X Fcycle 

7.911 

2 X PCG 

0.007 

2 X PCG 

0.012 

2 X PCG 

0.039 

5 X PCG 

0.026 

5 X PCG 

0.043 

5 X PCG 

0.104 

10 X PCG 

0.037 

10 X PCG 

0.072 

10 X PCG 

0.183 

10 X Smth 

0.105 

10 X Smth 

0.147 

10 X Smth 

0.249 

20 X Smth 

0.172 

20 X Smth 

0.236 

20 X Smth 

0.387 

30 X Smth 

0.221 

30 X Smth 

0.298 

30 X Smth 

0.482 

40 X Smth 

0.260 

40 X Smth 

0.346 

40 X Smth 

0.554 

50 X Smth 

0.291 

50 X Smth 

0.384 

50 X Smth 

0.611 
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